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ABSTRACT . ' — 

.One of the most -critical elements in the adaptation 
of productivity analysis to education* is the adequacy of the 
instruments for measuring program output or, achievement . various 
aspects of standardized test construction make them less than 
desirable measures of productivity. The Test Development Notebook 
vhich the New York State Department of Education is currently 
developing may offer a needed alternative to standardized testing. 
The overall purpose of the notebook is to provide sufficient numbers 
of test items at various ievel's of difficulty so that reading 
development at a given level may pe monitored periodically with the 

'administration^ of a series of comparable test forms. The starting 
point for the notebook has 1)een literal comprehension, and the work 
in process has two empixases: one is applying the cloze technique in 
testing literal comprehension; the other is systematizing the writing 
of muXtiple^choice items for literal comprehension by specifying the 

-type of questions to be asked and establishing rules for writing 
them. Both approaches use readability formulas to identify the 
difficulty of the material. Both stress objectivity and the 
elimination of writer biap. {A\ithor/EC) 
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A paper. presented at a symposium entitled "Productivity in 
R6adijig Programs" held during the annual meeting of the American 
Educational Research Association, Washington, D.C. from 
«arch 31, 1975 to April 3, 1975. T.ss.^ 7//4 



The initial purpose of this 'paper was to describe the varioiis materials 
and methods \ised'in the pilot study of productivity in reading programs. 

Preceding papers in this sympo3i\m have given considerable informaticm • 
on the instruments used for gathering data on the inputs, throughputs, pid 
output? of the pilot study. These instruments included (l) the schedules 

- . ^ r 

for th^ hour-long, taped interviews with pr^incipals, teachers, readijlg 
specialists, -and^aidea; (2) the forms for collecting data frm the School 
records of participating students? and (3) the tests, standardize^ and 
criterion-referenced; \us^ to assess achievement or output. This paper will, 
therefore, coiicentrate on th^^'effprts to improve measures for the assessment 
of outcomes arid will describe the^l^g^km way on the Test Development 
Notebook. ; , 

'J ' ' 

One of the most .critical elements in the Miration of productivity analy- 

sis to education is the adequacy of the instruments for m^i^ing program output 

or achievement. While inar^r^of tne res\lLts reported in this' sy^osium~on 

productivity in reading are based on standardized test, scoa^es, there is some 

question about the suitability of these measures. Various/. aspects of stand-. 

ardized test construction make them lej3s than desirable measures of ^oductivity. 

For &cample, wide ranges of difficulty are covered hy a single test, and ' 

items which ma.y test the primary goals of the reading program but do not , 

discriminate apionA pupils to produce the necessary distribution of scores, . 

? - \ 

are usually esdclud^d. Moreover, because test writing practices indicate a high^ 
degree of subjectivity in the selection of teBt content and in the writing oj 
reading tdst items, Ahere is no assurance that. the standardized test is an 



/^dequat^ , representation of the domain of reading. 

The Test Development* Notebook which the New York State Education , 
Department is currently developing may offer the needed alternative to 
stanjiardized testiiig. This work is an outgrowth of the development of the 
criterion-referenced CAM tests for the pilot phase of the productivity study. 
The overall purpose of the notebook is to provide sufficient numbers of 
test items at various levels of difficulty so that reading development at . 
a feiven level may be monitored periodically with the administration of a 
series of comparable test forms. 

The Notebook in its finai' form will be a set of master copies of test 
items on, various aspects of reading that can be assembled in a variety of 
ways and then reproduced in quantity to meet local needs and local test 
Bpecif icatipns . The starting point for the notebook has been literal g^o^^ 
prehension, and the work in process has two emphases: one .is applying the 
Cloze Techniqiie in testing literal comprehension; the other is system'itizing 
the writing of multiple-choice items for literal comprehension by specifying 
the types of questions to be asked and establishing rules for writing them* 
Both approaches use readability formulas to identify the difficulty of the 
material. Both stress objectivity and the elimination of writer bias. 

Readability Scores and Difficulty Levels 
One need in reading assessment, and in reading instruction as well, is to 

have material at an appropriate level of difficulty. For assessment purposes.^ 

multiple passages of comparable difficulty are required so that progress 

may be measured at a given level. 

The perfect readability formula accounting for ^JLl possible variables 

has yet to be devised, but existing formulas do permit one to.yank reading 
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materials on their "difficulty" as defined hy the formula used. 

The readability -formulas used for the Test Development Notebook are the 
Spache (1953, I960) and the Dale-Chall The Spache is normally used 

for grades 1 through 3, the ^ale-Chall^for grades 4 through 12 and college. 
Botl> formulas use average sentence length and percent of "hard words" in figuring 
difficlilty. "Hard words" are those not appearing on lists of famiUar words. 
The word list for the Spache fomula is "Clarence ^tone 's Revision of the 
Dale List of .796 Easy Words "; for the Dale-Chall formula it is the "D61e 
Ust of 3000 Familiar Words." The criteria for difficulty used in devising 
both formulas were graded reading materials. The Spache formula produces 
grade level'scores. The Dale-Chall formula 'produces raw scores interpreted as 
"corrected .grade levels." The corrected grade level Ibr a raw score of 5.0 to 
5.9 on the Dale-Chali; for example, is 5th to 6th grade. 

For t^e Test Development Notebook-^ the pange of Spache scores was 
diHded into. 6 equal intervals, and the range of Dale-Chall scores was" 
divided into 22 equal intervals. This gave 28 difficulty levels covering 
^ades 1 through college. The raw scores, difficulty levels-, and original 
g^ade level interpretations are shown in Table 1." 



Insert Table 1 about here 



Appljring the Cloze Technique ^in 
' Testing Literal Comprehension ~ ^ 

The cloze technique— the systematic deletion of words from prose 
passages and their subsequent repOacement by the testee~is best known as 
a mean^ of determining the readability of written material for specific 
pupils. It has been found (Bormuth, 1971) that students must attain a 
certain proportion of correct completions (35^-50^) if the material from 
which a.clozed passage is taken is to be useful to them in instruction. 
It has also been found (Bormuth, 196?) that scores on the cloze test correlate 
positively with gain scores on tests given Before and after the reading of 
a passage. 

Some theorists regard information gain as the most acceptable operatipnal 
def initial of literal reading comprehension. If one accepts 4^ormation gain 
as a- sound indicator of comprehension, the cloz^ technique may be, a quick 
^ and economical way of measuring comprehension. Accordingly cloze item'i 
are incorporated in the Test Development Notebook for two purposes: (l) 
to provide a method of determining pupils' reading levels, and (2) to assess 

literal comprehension. 

^ . . , , — ^ _ ^ 

In its most common form, the cloze technique calls ^or the student to 
supply the deleted words: in the Test Development Notebook, cloze passages 
will be followed ^.multiple choice responses suitable/for machine scoring. 
Selection of Cloze Passages / 

In order to eliminate the possibility of contei^t bias and to assure 
that the cloze tests would feflect the materials a/udents read in instructional 
programs and .elsewhere, a plan was devised for random sampling of four domains: 



1* Textual Material in Reading, Language, Arts, 
-Social Studies, Science, an^( Mathematics 



2. Citizen Material, i«e,, newspapers and news 
magazines 

3. Consumer Material (catalogs, j advertising, 
instructions, and so forth) 



4* Reference Materials Ctest instructions, children's 
magazines, encyclopedias, and so forth; 

The' sources of these materials were the New York State Education Depart- 
ment 's Curriculum laboratory and the State Library. Some consumer passages 
were taken from The Assessment of Student Literacy (Hanson and Hesse, 
1974) used in the Madison, Wisconsin^ Public Schools. 

The san^jles taken were examined to Iqcate coherentr- passages of specified 
lengths appropriate for dozing. Readability scores were then calculated and 
^ the passages assigned to grade levels. 

Originally the cloze passages were, to ba grouped by dif f iculty JLevels 
without reference to grade levels. However, the range of difficulty in textual 
material for any one grade was so great that grade level groupings for cloze 
passages were adopted as more usefu^ to teachers. In the textual domain,* 
the number of difficulty levels pei; grade level varies. In the three other 
domains, grade levels are limited/to two diffic\ilty levels. 

TJie selection procedures produced 1,374 passages for cloze testing. 

/ 

Their distribution by domain ahd grade level is shown in T^ible 2. Table 2 

/ 

also shows the distribution of the^ trextual materials by subject matter. 
' Each cl02j'e passage has a code number giving its grade level, difficulty level, 
material domain, and any subcategories to whidh it belongs. 
Preparation of Cloze Items 

The procedure for word deletion in the cloze passages varied with grade. 
For grades 1 and 2 every 8th wbrd was deleted. Deletions were lijnited 



to nouns and verbs. For grade g and above, every 5th wprd \*as deleted. 
Deletions included adverbs and adjectives as well as nouns and verbs. 

In all cases the initial deletion was made between the 6th and the 
10th words. The exact starting point was determined by a table of random 
numbers. The number of deletions per passage was fixed by the passage 
leng:th which varied by grade level. The number of alternatives in the 
multiple-choice responses also varied by grade level: three alternatives 
at grade 1, four at gr^de 2, and five at grade 3 and above.' These var- 
iations by grade level are summarized in Table 3, Specifications for Cloze 
Passages and Test Items. 



Insert Table 3 about here 

The correct multiple-choice response to a cloze item is the exact word 
deleted from the passage/ To assure distractors of appropriate difficulty 
for the test items, graded lists of nouns, ^yerbs, adverbs, and adjectives ^ 
were prepared using Harris and Jacobson's Basic Elementary Readiniz Vocabularies 
(1972) and EDL^Research and Information Bulletin 5: A Revised Core Vocabulary 
(Taylor, Frackenpohl, and White, I969). Special content lists for subject 
matter areas were conq^iled using the Haf3?is-Jacobson material and the 
American Heritage W ord Frequency Book (Carroll, Davies, and Richman, 1971). 

> Initially, distractors were selecited from appropriate lists by use of 
a table of random numbers. Later, a computer program was written for. automatic 
random selection of distractors. Each set of distractors was reviewed to 
eliminate tricky or ineffective distractors, such as synonyms, and to assure 
that the distractors- agreed with the stem in tense, numbor, and so fortri. 

\ 

r' 
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With a minirmnn of 3 deletions per passage at grade" 1 and a maximum 
of 10 deletions per passage at grade 4 and above, nearly 13,000 multiple- 
choice items have been prepared for the cloze section of the Te^t Development 
Notebook. Their distribution by donain is shown in Table 4. 



Insert Table 4 about here 

Format 

All cloze passages and test items have been put in a comparable format. 
The format gives (l) the identification number of the passage, (2) a title 
(provided by the itein writer), (3) the passage itself, and (4) the test items. 
Large (Bulletin) type is used for the first two grades. Grade 1 passages 
appear two to a page. At all other grade levels, each passage and its items 
appear on a separate page. The test items •aaTe not numbered so that teachers ^ 
may assemble the pages in different ways for different tests and then number 
the items consecutively. Sample cloze pages for grades 1 and 4 are shown in 

Figure^ 1 and 2. 

Insert Figures 1 & 2 about here 
Future Work ■ ^ : 

Plans have been made to extyend the clo ze ma terial in the textual domain 
through college and to expand the number of passages for the citizen, consumer, 
and reference domains. 

In the meantime, the completed test items are already in use on a trial . 
basis. Users are well aware that this application of the cioze as a measure 
of literal comprehension is experimental. Various internal analyses and 
compal?isons wfth 'other tests must be .made to determine its utility and 
validity. Some of these analy sea will examine the use of the multiple-choice- 
fonnat for a cloze test. 

-7- . ^ 
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Syst emati zing the Writing of Multlple*>Choice Items 

Multiple-choice items based on given selections are probably the most 
commonly used means of assessing comprehension in school reading programs. 
Systematizing the writing of multiple-choice items for the Test Developiaent 
Notebook is intended to minimize the idiosyncracies and biases of individual 
item writers, and to maximize the comparability of items for the same skills 
or objectives. With more comparable items one can better trace students' 
progress at a given level and from level to level. 

K The systematization process — specifying types of questions and rules for 
writing theli| — may even*tually be extended to all aspects of comprehension. . 
The work nearing completion at this time is focused on literal comprehension ^ 
and, mor^ specifically, on main ideas and details. The definition 
of literal comprehension used for the Test Development Notebook is the ability 
to identify or recognize exactly what is said in written material. This may 
'be regarded by, some as too simple a definition cf comprehension, but accuracy 
in knowing what has been stated is fundamental to any interpretation of the 
material. 

The work on the multiple-choice items has had two* phases. The first 

m 

involved the preparation of over 300 passages meeting specified requirements, the 
second the actual writing ,of main idea' and detail questions for each passage. 
Passage Preparation 

The design for the literal con^jrehension section of the Test Development 
Notebook called for 15 passages for each of ,the first 20 difficulty levels 
established from the Spache and Dale-Chall formulas. The lengths of the 
passages were to vary by level. The 'specified passage lengths and the read- 
ability scores for all 20 levels are ^ shown in Table 5. Additional specifications 
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Ihsert Table 5 about here 

concerned the unity of each passage, its utility as a source for main idea 

and detail questions, and its suitability in content, style, and vocabulary 

for the pupils with whom it would normally be'used* The vocabulary of the 

passages. was controlled to a great extent by the word lists of the reachability 

formulas- Vocabulary was- further tontr<)lled by the use of Harris and Jacobson 

* 

(1972) basal-reader or "core" word lists for levels 1-12 (grades 1-6) an|4 
the American Heritage Word Frequency Book (1974) for levels 13-20, TheL 
references served as guides for determining the acceptability of indivi'duaaL. 
words in passages and in it'em responses- 

Passage material was taken from existing criterion-referenced tests 
(the Duval County, Florida, tests for Individually Paced Instruction in 
Reading and the CAM tests used in the pilot phase of the study) and from a 
variety of books, magazines, and newspapers, A substantial lanount of new 
material was written. Existing test passages were edited extensively to meet * 
the passage specifications. Modifications in excerpts from books, magazines, 

vocabulary requirements of the ^readability formulas. An effort was made to 
have a balance of^ fictional and non-fictional passages and to have diversity 
of subject matter within these broad categories. Fiction was given greater 
weight at the lowe^ levels. The final distribution of the passages by topic* 
and level is shown in Table 6. 



and newspapers were limited to a few individual word^hanges ^0 meet the*" 



Insert Table 6 about here 



In, systematizing the actual writing of test itemd^, 12 different types of 
questions were identified; 4 for main idea and 8 for details. 



M^ii^ Idea questions . Main idea questions concerned titles (often used 
as devices ibr teaching main ideas) and main ideas" themselves. For the 
purpose of item-writing, a ttt3e was defined as a noun or noun with, modifiers 
which expresses the subject or topic of a passage. A main idea was define 
as a complete sentence incorporating the essential point or points of * 
passage. Two typfes of title^and main idea items were written: (l) verbatim 
items with the correct responses and the disti^ctors taken directly from the 
passage, without change/ and (2) derived itpns that used the vocabulary of 
the passage for the correct response and distractors but ntedif led word qrder, 
added words, and consolidated mai^erial as needed. The derived items were 
to adhere to the literal meaiiLng o^ the passages and to avoid inference or 
abstraction. ' . ' ^ 

Verbatim and derived items l^ave identical stems. The exact w<^ding of 
the stems and the number of alternatives vfii^ with the level of the passage' ; 
as shown in Table 7. 



Insert Table 7 about here 



While standardized^ in format, the title and main idea questions do rely 
heaviXj^-'ori writer judgement for de1>erminin^ tha titife and main idea. 

^ • ' ■ . 

Uetail questions. Tne eight detail questions were suggested by Bormuth's 
wprk On the Theory of Achievement Test' Items (19701 Bormuth identifies 8 wh^trans 
formations capable of^ covering Ifehe entire lexical content oi^ a sentence. The 
"pro" or substitution el^ents <^hich can transform a statement into a question 
are how> -« ^hat (for a npun'or pronoun); what (for. a verb), which, who, wheji . 
where ^ and wh y > These are, of course, the quesijion words teachers commonly 
use to direct attention to the details of reading materials ^ 
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Bopnuth proposed that for test construction purpose^ a pool of items be 
developed by writing all possible whsLtems on a sentence and on all possible 
transfonnat^ions of and derivations from that sentence. An individual test would 
be created by randan selection of items from 5uch a pool, , The ability to 
handle, all transformations and derivations is>beyond the scope of the literal 
congjrehension sectioh of ^ the Test Development Notebook, but the wh-substitutioxB 
established the pattern for the detail questions. 

i 

Detail questions for a gi^ven passage were written as follows: 

1. A permutation table for numbers 1 through l6 was 
used to take sentences. from a passage in random 
order. ^ ^ 

^. Where possible, a how question was written oh 
the first sentence taken, a what (nouri ques- 
^ tion on the second sentience, a what (verb) 

question on the third sentence, and so forth 
- - - through the entire list of wh-transf ormations . 



3. If this, first sentaice was not appropriate for 
a ^dwj^'^cluestion, its utility for each succeedJjng 
vrti question \yas "examined, and an appropriate jjh 
^ item writtea. 

4.. Each randomly selected sentence ^s treated in \ 

th6 same manner until one test item was written, * 
for as many of the wJi questions as possible. 
If necessary, a single sentence was ufied for ' * 
more than one question. " * . 

' The eight different detail or j^ii-qxlestions are l^^ted in Table 8 with 

sample sentences, questiorjp, and responses. 



Insert Table 8 about here 



The rules for question-writing were: 

1. Be "clear and concise. 

2. Use colloquial English, 



3. Change the portion of the sentence 
appearing in the question as little 
as possible. 

4* Replace pronouns vdth their referenta. 
5. Avoid negative question?. • 

\» 
6. Always start the question vdth the 

"pro" element, ip., how^ what^ ' ^ 
and so forth. 

The rules for writing responses were: 

1. Take the correct response verbatim 
from the passage. 

2. Make all distractors grammaticaJJLy, 
semantically, -and logically plausible. 

3. Have distractors parallel in length and 
construction. 

4. Do not include -distractors that could be 
considered correct on the basis of other 
sentences in the passage. 

5. Take distractors from the passage where- 
ever possible, modifying wording to meet 
the preceding qualifications. 

6. Make up distractors only vrfien necessary, 
and have them conform to the preceding 
qualifications (2, 3^ 4 above). 

The overridiig criterion for every item was that it be impossible to 

answer without reading the passage. 

The rules for item-writing automatioally excluded questions on negative 
statements, questions, ija^Derative statements, sentence fragments, ellipses, 
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and many instances of anaphora (the use of a grammatical substitute for 
a preceding word'or group of words). Long compound sentences, predicate 
adjectives, and passive constructions were difficult to handle because' they 
often produced unduly conqjle;}^ awkward or uncolbquial questions. One ques- 
tion that was consistently avoided as uncolloquial • was the vrtw 
question based on a sentence with a plural subjects and a present tense verb. 
Transforming "The men a&x4 the bridge is out^' to 'Who say the bridge is 
out?". was not considered acceptable. 

The limiting effects of the rules were most apparent with upper level 
passages where longer sentences, more complex ideas and complicated sentence 
structure, ellipsis^ and anaphora were most likely to occur. On the other 
hand, passages at the lowest, difficulty levels often were too brief or devoid 
of modifier^ to produce "the "full array "of ijh questions. 

With eight vrtj questions plus the four main idea and title questions, the 

maximum number of items that could have been written for the 300 passages 

was 3,600. Because all questions could not be asked on every passage, tne 

number produced was closer to 3000. Figures 3 and 4 are exaasplea of final 

products for levels 2 and 9. - Z m" "~„ , 

^ Insert Figures 3 & 4 about here 
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Future Work; The literal comprehension* questions will be field tested this 
spring along with the cloze items. 

The multiple-choice materials will be sub3ect to much scrutiny with an 
effort made to ascertain pupil M teacKer,^eaction to both the passages and 
the questions. One indicator of the. effectiveness of the passage preparation 
and Uem writing 'procedures will be what passages and questions are chosen 
for use. Comparisons of scores on different pa'asages f^om the same level 
may give some indication of the efficacy of readability formulas for pro- 
ducing -passages of comparable difficulty. Test results' for the these literal 
comprehension, multiple-choice items will also be compared with results on 
the cloze tests and standardized tests. ' 

sill' 
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Oifficu*Lty Levels for Test Development Notebbok 
***** \ • ■ » / 



Rfiadabilitv 
Formula 


Raw 


— ' Vf- 

Score \ 


Difficultv 
Level 


Original Grade 
Level Assigninent • 




' 1. 0 




,1. 4 ' \ 






p 


!• 5 




1. 9. 
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2 / : 

/ 


1 


A' 


2. 0 




2. 4 1 
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2* 9 4 
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D 
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- 


^ 

4 7'4» 
4.99' 
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■ ' ' 8 


4 


A 


5.00 


- 


5.24 


■ 9 




I 
£ 


5.25 
5.50 


- 


5.49 
5.74 


10 

• ir ■ 


5-6 
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5.75 




D.99 


12 
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H • 


6.00 
6.25 




6 24 
6.49 
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6.75 
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7.24 


■ •* 17 






7.25' 




7.49 


5, 18 
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7.50 
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- 


7.99 


20 




. S 
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8.50 
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8.49 
8.74 
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25 






•9.25 




9.49 
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13-15 




9.50 




9.74 


27 
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28 
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Specifications for Cloze Passages 
*/and Test Items 

/ 






• Grade 1 


Grade 2 




Grade 3 


Grade 4 
and Above 




r 

Passage, 
LengtTi 


25-35 
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40-45 
words 
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words 
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words 


Words 
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Nouns 
verbs 
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verbs 
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Verbs 
Adverbs 
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Verbs 
Adverbs 
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of Dele- 
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8th 
Word 
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8th 

Word 
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5 th 
Word 

4 
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5 th 
Word 


Deletions 
per .Pass- 
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3 


5 




10 


10 


Distractors 
per Item 


3 


4 




5 

k 


— P — 
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I 
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Figure 1 

« 

Sample Cloze Item Level 1 



THE BIRDS AND -BEN 



r 



Ben comes ^0 see the birds, 
fly up. The birds fly down. They 
Ben says, "Fly, birds." The 



The 



They fly up high. They fly aviay. 



to Sen, 
fly up. 



Q a. paints 

b. birds 

c. pigs 



O a- ^^^^ 
b. come 

C; put 



O 



a. men 

b. browns 

c. birds 



V 



01 
-i- 


01 


01 




304 



ninK WAfiTS HELP 



* Dick said, "Come here, Jane. I . 

you. Com^'and help me." . n.'. a 

^ S^ly , "Oh, vJanel See Dick and 

Spot!" 



you." 



"Here! Dick," said Jane. "I can 



•Q a. name ^' a. had 

b. surprise 6. said 

c. wcfnt c. raced 



O a. truck 

b. help 

c. picture 
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Figure 2 
Sample Cloze Item Level 4 



. JOEY WALKS HOME FROM SCHOOL 



Joey had a certain way he walked home from 

. He did not walk up street and * 

out. 



down the 
He could 



. He had it all 

cater--corners across streets. He could 

from the south-east of this one 



to the north-west 



of" that one. He 



that he saved at least one 



iblock, by walking home from school this wayi 



o 



o 



o 



o 



o 



a. 


insect 


b. 


momma 


c. 


prairie 


d. 


canoe 


e . 


school 


a. 


weak 


b. 


one 


c. 


smart 


d. 


yellow 


e. 


grand 


» 

A. 


next 


b. 


round 


c. 


dumb 


d. 


high 


6. 


picture 


a. 


minded 


b. 


alerted 


c. 


topped 


d. 


figured 


e. 


buzzed 


a . 


mop 


b. 


kid 


c. 


charge . 


d. 


yell 


e. 


cut 



o 



o 



o 



o 



o 
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a. 


doubt 


b. 


push 


c. 


steal 


d. 


walk 


e; 


park 


a. 


gathering 


b. 


grain 


c. 


corner 


d. 


ocean 


e. 


packet 


a. 


notice 


b. 


jacket 


c. 


corner 


d. 


instrument 


e. 


heading 


a. 


fired 


b. 


cracked 


c. 


arose 


d. 


choked 


e. 


knew 


a. 


outer 


b. 


faint 


c. 


solemn 


d. 


homely 


e . 


whole 



' • ■ . Table 5 . 

I 

* ' Length and Readability Score Specifications • 

for Literal Comprehension Passages 



Level Words Readability Score 



1 


26 • 


35 




1. 0 


- 1.4 


2 


36 - 


45 




1^ 5 


- 1.9 


3 


46 - 


55 




2» 0 


- 2.4 


4 


56 - 


65 




• > 
2. 5 


> 

- 2.9 ' 


5 


^ 65 - 


75 




. 3. 0 


- 3.4 


6 


76 - 


85 ' 




3. 5 


- 3.9 


7 


> 86 - 


95 




4.50 


-4.74 


8 ' 


9p - 


105 




4.75 


- 4.99 


9 


' '106 - 


115. 




5.00 


-5.24 


i'O " , 


116' - 


,125* 




,5.25 


-5.49 


11 


• . l26 - 


135 




5.50 


- 5.74 




136 - 










12 


145 




5.75 


- 5.99 


1*3 








6.00 




•14 


156 - 


165 




6.25 


- 6.49 


15 


166 - 
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6.50 


- 6.74 


16 ' >• 


r66 -/ 


175 




6.75 


- 6.99 


17 


; ' , 166 - 


220* 




7.00 


- 7.24 


1^ 


166^-" 


220* 




7.25 


- 7.49 


19 


166 


220* 




7.50 


- 7.74 










^.75 




20 


166 - 


220* 




- 7.99 



* At the four highest levels , the word range was extended in order to have 
fictional passages v/iyi the required readability scores. Non-fictional 
passages were h^ld to; a maximum of 185 words. 
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Table 7 

Title and Main Idea Questions 



Question // Type 



Passag^ 
_Levfel_V 



Stem 



Alternatives 



1 Title-Verbatim 

2 Title-Derived 



1-6 ^ -^^^ The bes^ title for 
this^story is 

7-20 The be^t title for 
this selection is 



Main Idea- 
Verbatim 



Main Idea- 
Derived 



1-4 What i%. thi53 story 

mostly about? 

5-6 The main idea of this 

story is 

7-20 The main idea of this 
selection is 
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Figure 3 , , • 

Literal Comprehension Passage 
and Items for Level 2 

A' big brpwn dog lives n-fext^door to Bruce, Hi^ name is Sparky^ 
He barks a lot^ Every time Bruce comes near, Sparky barks* His bark 
• is very loud# .Bruce does not like to go near the big dog. 

1, The best title for this story is 
X, a) Sparky 

b) ■ Dmoc T"/»i^ . ^ 

c) Next Door 

2, Th/best title for this^ story is / 

a) A Big Brown Dog ^ 
X b) Bruce and Sparky ' * 
c) Bruce and the Bark • 

3* Wi^at is this story mostly about? 

a) A big brown dog lives next door to Bruce, 

b) Every time Bruce como near. Sparky barks ^ 

X c) Bruce does not like Do go near the big dog, ^ 

s 

4. Whatrt«>^this story mostly about? 

a) Bruce loves Sparky, 
X b) Bruce doe^ not like Sparky, 
'-^^ ^ c) Bruce dbes .not like to see Sparky, 

' 5« How much dods Sparky bark? (3)* 

a) a little » ' 
X b) a lot ' 

c) not too much ' * ^ ^ • 

6. ^at is very loud? (5) 

X a')^ Sparky* 6 bark 

b) Bjuce' s bark 

c) Spot* s bark 

* Numbers in ^parentheses after questions indicate source sentence in|*passage. 
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i What does Sparky do every time Bruce comes near? 



X- |0 barks 
bK runs 
c) \sits 



When does Sparky bark? (A) 

a) every time a cat comes near 

b) every time a dog comes near. 

c) every cirae Bruce comes near 



re ioes Sparky live? (1) 



a) next door to Bruce 

b) next door to B'sn 

c) down the street 



Which dog lives next door to Bruce? (1) 

a) a big black dog 

b) a big browT> dog* 

c) a big white dog 

Whe does- not like to go near the big dog? (6) 

i 

a) Spark:; 

b) Bruce 

c) Ben 




Figure A 

Literal Comprehension Passage 
. and Items for Level 9 



Mike works in the deiaolition business. -He helps knock down old 
buildings so new ones can be built* Sometimes Mike drives a huge crame 
thatf^swr^gs a heavy ball* When the ball smashes against the wall of a 
building/ the, wall always cr\ini3les* 

Mike enjoys his work very much. He often thinks- about his work. 
And he talks about it to anyone who will listen. Sometimes Mike says 
he has demolition in his blood. Sometimes his wife thinks he has 
demolition on the brain. When Mike drives home from work, he crashes 
his car into the garage wall. In his living-room^ Mike trips and 

r 

demolishes the coffee table. 



1. ,The best title for this selection is 

a) ^ The Demolition Business 

b) Against the Wall 

X c) Demolition in His Blood 
d) In the Living Room 



2^ The best^title for this selection is 

X a) Mifee the Demolition Man 

6) Crumbling Walls 

c) Enjoying One's Work- 

d) The Car Crash 



3. The main idea of this selection is 



a) Mike helps knock down old buildings so new ones can be built < 

b) The wall alyays crumbles* . 

X c) Mike enjoys his work very much.. 

d) Mike crashes his car into the garage wall. 



4. The main idea of this selection 

a) The swinging ball always crumbles a wall. 

X b) M,*.ke is. always demolishing things, on his job and at home. 

c) In demolition something is wrong with the blood or the brain, 

d) People who enjoy'^their work are always thinking about it. 



ao 



y\ How often does Mike drive a huge crane? (3)* 

a) every cjay 

b) seldom 

X c) sbmeti^nes 
d) never 



6. What always crumbles? (4) 

• a) the crane ' , 

b) . the wall 

c) the car , 

d) the coffee table 

7. What do«s Mike do in his living room? (11) 

J \ y - ' • • 

^ a) works 

b) thinks 

c) .talks 

• X d) trips 

6. When does Mike crash his car into the garage wall? (10) ' 

a) when he helps knock doWn old buildings 

b) When he thinks about his. work 

c) . when he enjoys his work 

X when he drives #hone from work. 

9. Where does Mike's wife sometimes think he has demolition? 

a) ifi his business • \_ 

b) in hi? blood- - ; 
X c) on the brain 

d) in the living room \ 

10. Which business does Mike work in? (1) 

a) building 

b) garage 

X c) demolition . , , 
d) .coffee ^ * 



Who ehjo-ys his work very much? (5) 



a) Mack 

X b) Mike 

• c) the driver 

d) Mr. Crane 



* Number In parentheses after question irjd lea te^ source sentence in passage. 



gp^i^" * Number in parentheses atter question 



4 



12* ' Why does Mike help knock down old buildings? (2) 

a) because he drives a huge crane 

b) so he can swing a heavy ball , 
X c) so new buildings can be built 

d) because he has demolition pn the brain ✓ 
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